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ABSTRACT 

Virtual reality technology offers the promise of 
interaction with a computer-based environment that engages visual, 
auditory, and tactile perception. Three interrelated virtual reality 
design topics are particularly relevant to visual literacy. The first 
is the concept of avatars. Avatars are agents that appear in a 
virtual world representing the user. The second design topic is 
affordances. According to the theory of ecological psychology of J. 
J. Gibson, affordances are the distinctive features of a thing that 
help distinguish it from other things that it is not. As such, they 
provide strong clues to the operations of things. The third design 
topic is interfaces. Interfaces for both input and output link the 
user to the virtual world that exists only digitally. Examples are 
given of the use of each of these design topics in curent computer 
games and in the science fiction novel "Snow Crash" (N. Stephenson). 
These interrelated topics must be considered by designers who explore 
the potential of the new medium. (Contains 13 references.) (SLD) 
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Introduction 

Virtual reality technology offers 
the possibility of becoming immersed 
in and interacting with a computer- 
based environment that engages 
visual, auditory, and tactile percep- 
tion. Building on the fields of 
computer graphics, computer-aided 
design, human-computer interfaces 
and simulation, virtual reality was 
devised to enable people to deal with 
information more easily. Virtual 
reality is potentially an extremely 
versatile and powerful creative tool. 

Virtual reality is currently in the 
very early stages of development. 
The practical potential of this 
technology is still under exploration, 
but it appears to offer important 
educational potentials in the 
following areas: (1) modeling 
complex phenomena, (2) project 
planning and design, (3) the design of 
interactive forms of entertainment 
and learning, (4) communication and 
control at a distance, (5) virtual field 
trips, and (6) the design of 
experiential learning environments. 
In addition, virtual reality offers 
many possibilities as a tool for 



nontraditional learners, including the 
physically disabled and those 
undergoing rehabilitation who must 
learn (or relearn) communication and 
psychomotor skills. Virtual reality 
offers applications in many fields, 
including robotics, medicine, 
rehabilitation, scientific visualization, 
aviation, business, architectural and 
interior design, city planning, product 
design, military training simulations, 
law enforcement, entertainment, the 
visual arts, music, dance, communica- 
tion and collaboration at a distance, 
and education. 

In thinking about virtual reality, 
immersive technologies involving 
wired gloves and head-mounted 
displays first come to mind. But in 
addition to immersive VR, there are 
several other types of virtual reality, 
including mirror worlds, desktop VR 
(also known as 'through the window 
VR'), cab environments, Waldo World 
VR, and the CAVE. 

In this paper, the focus will be 
on three inter-related VR design 
topics: (1) avatars; (2) affordances; 
and (3) interfaces. These topics are 
particularly relevant to visual literacy. 



Avatars 

Avatars are agents that appear in 
a virtual world representing the user. 
For example, in many VR systems, an 
image of a hand appears to represent 
the user's hand encased in a wired 
glove. Thus the user can see where 
her hand is in relation to the other 
objects in the virtual world. More 
sophisticated avatars can be seen in 
virtual reality games such as Dactyl 
Nightmare and Legend Quest. 

To understand what an avatar is, 
think of Rene Magritte's painting of a 
pipe that included the words "Ced ce 
n'est pas un pipe" (This is not a pipe). 
Magritte was playing with our under- 
standing of reality and representation. 
With the introduction of avatars in 
virtual reality, representations 
(avatars) become dynamic and seam- 
lessly linked to reality, in real time. 
With virtual reality, reality and 
representation are ever more closely 
intertwined. 

The Legend Quest game hints at 
just how closely reality and represen- 
tation can become merged in virtual 
reality. Based upon the Dungeons & 
Dragons role-playing game, Legend 
Quest uses four immersive virtual 
reality systems networked together so 
that four players share the same 
virtual world, which includes a forest, 
a castle, and a multilevel dungeon. 
Players are able to see and speak with 
each other. A special voice 
input/output system allows them to 
speak in the voice of the character 
they have chosen to play. Thus the 
avatar is not only visual but auditory. 
Each player sees and hears a repre- 
sentation of his or her character 



within the virtual world. There are 
eighteen possible characters. The 
possibilities are limited to an Elf, 
Dwarf, or Human of either sex who 
may be a Wizard, a Warrior, or a 
Thief. Every aspect of character has 
disadvantages as well as advantages, 
so a team with the most varied 
aspects and talents is the most likely 
to succeed in the quest. All four 
players are on a quest together, and 
the dynamics of the game create the 
most success when they cooperate 
with each other. The goal of the game 
is to master ten challenges while not 
dying in a hazard or from foul play. 
The programming of the monsters in 
the dungeon uses artificial intelli- 
gence that allows them to learn from 
experience and make countermoves. 
Winning is not easy (Hamit, 1993). 

Another example of an avatar 
comes from Waldo World VR applica- 
tions. This type of virtual reality is a 
form of digital puppetry involving 
real-time computer animation. The 
name "Waldo" is drawn from a 
science fiction story by Robert 
Heinlein (1965). Wearing an elec- 
tronic mask or body armor equipped 
with sensors that detect motion, a 
puppeteer controls, in real-time, a 
computer animation figure on a 
screen. 

One example of a Waldo World 
VR application is the Virtual Actors™ 
developed by SimGraphics 
Engineering. These are computer- 
generated animated characters con- 
trolled by human actors, in real-time. 
To perform a Virtual Actor (VA), an 
actor wears a "Waldo" which tracks 
the actors' eye brows, cheek, head, 
chin, and lip movements, allowing 



them to control the corresponding 
features of the computer-generated 
character with their own movements. 
The animated character is a form of 
avatar that makes subtle movements 
and expressions based on the corre- 
sponding gestures and expressions of 
the puppeteer. Here the avatar is at 
center stage. A hidden video camera 
aimed at the audience is fed into a 
video monitor backstage so that the 
actor can see the audience and 
"speak" to individual members of the 
audience through the lip-synced 
computer animation image of the 
character on the display screen. This 
digital puppetry application is like 
the Wizard of Oz interacting with 
Dorothy and her companions: "Pay 
no attention to that man behind the 
curtain!" 

The science fiction novel Snow 
Crash by Neal Stephenson plays with 
the notion of avatars in the Metaverse, 
the networked virtual world where 
much of his story takes place. In the 
Metaverse, it is possible to have an 
expensive, custom-designed avatar 
that is capable of subtle facial 
expressions or, for the less affluent, it 
is possible to get a mass-produced 
avatar from someplace like Walmart. 
These mass-produced avatars come 
with model names. For a woman, the 
low-cost avatar goes by the name 
"Brandy." Here is Stephenson's 
description: "Brandy has a limited 
repertoire of facial expressions: cute 
and pouty; cute and sultry; perky and 
interested; smiling and receptive; cuts 
and spacy. Her eyelashes are half an 
inch long, and the software is so 
cheap that they are rendered as solid 
ebony chips. When a Brandy flutters 
her eyelashes, you can almost feel the 



breeze" (p. 37). The male counterpart 
is a "Clint" A Clint avatar "is just the 
male counterpart of Brandy. He is 
craggy and handsome and has an 
extremely limited range of facial 
expressions." There is also a low-cost 
"Avatar Construction Set" for the 
more creative. 

In the Metaverse, avatars offer 
certain advantages: "Your avatar can 
look any way you want it to, up to the 
limitations of your equipment. If 
you're ugly, you can make your 
avatar beautiful. If you've just gotten 
out of bed, your avatar can still be 
wearing beautiful clothes and profes- 
sionally applied makeup" (p. 36). 
Celebrities can spend a night on the 
town in the Metaverse without fear of 
robbfeiy, violence, or bother from 
over-enthusiastic fans. 

Affordances 

Let's now turn to the second 
design topic, affordances. 
Psychologist J.J. Gibson (1986) has 
proposed a theory of ecological 
psychology that emphasizes human 
perception of the environment, 
specifically the psychology of the 
awareness and activities of an 
individual in an environment. In 
Gibson's theory, "affordances" are the 
distinctive features of a thing which 
help to distinguish it from other 
things mat it is not. Affordances help 
us to perceive and understand how to 
interact with an object. For example, 
a handle helps us to understand that a 
cup affords being picked up. A 
handle tells us where to grab a tool 
such as a saw. Affordances provide 
strong clues to the operations of 
things. 



Affordance perceptions allow 
learners to identify information 
through the recognition of relation- 
ships among objects or contextual 
conditions. Affordance recognition 
must be understood as a contextually 
sensitive activity for determining 
what will (most likely) be paid 
attention to and whether an 
affordance will be perceived. The 
ability to recognize affordances is a 
selective process related to the 
individual's ability to attend to and 
learn from contextual information. 

Gibson's model of ecological 
perception emphasizes that percep- 
tion is an active process. Gibson does 
not view the different senses as mere 
producers of visual, auditory, tactual, 
or other sensations. Instead he 
regards them as active seeking 
mechanisms for looking, listening, 
touching, etc. Furthermore, Gibson 
emphasizes the importance of 
regarding the different perceptual 
systems as strongly inter-related, 
operating in tandem. Gibson argues 
that visual perception evolved in the 
context of the perceptual and motor 
systems, which constantly work to 
keep us upright, orient us in space, 
enable us to navigate and handle the 
world. Thus visual perception, 
involving head and eye movements, 
is frequently used to seek information 
for coordinating hand and body 
movements and maintaining balance. 
Similar active adjustments take place 
as one secures audio information with 
the ear and head system. 

Gibson hypothesized that by 
observing one's own capacity for 
visual, manipulative, and locomotor 
interaction with environments and 



objects, one perceives the meanings 
and the utility of environments and 
objects, which Gibson calls "affor- 
dances." Following from this, one can 
predict that the ways in which the 
user is allowed to interact with virtual 
things in a computer-generated world 
will determine how well he or she can 
understand them. Gibson's ideas 
highlight the importance of under- 
standing the kinds of interactions 
offered by real environments and the 
real objects in those environments. 
This knowledge from the real world 
can inform the design of interactions 
in the virtual environment so that 
they appear natural and realistic, or at 
least meaningful. 

How does this relate to virtual 
reality? According to Rheingold 
(1991), a glove that controls a virtual 
object would be an "affordance," a 
means of literally grabbing on to a 
virtual world and making it a part of 
our experience. Rheingold explains: 
"By sticking your hand out into space 
and seeing the hand's representation 
move in virtual space, then moving 
the virtual hand close to a virtual 
object, you are mapping the dimen- 
sions of the virtual world into your 
internal perception-structuring sys- 
tem" (p. 131). So the image of a hand 
in a virtual world is both an 
affordance and an avatar. 

Virtual reality pioneer Jaron 
Lanier (1992) has commented that the 
principle of head-tracking in virtual 
reality suggests that when we think 
about perception - in this case, sight 
— we shouldn't consider eyes as 
"cameras" that passively take in a 
scene. We should think of the eye as 
a kind of spy submarine moving 



around in space, gathering informa- 
tion. This creates a picture of percep- 
tion as an active activity, not a passive 
one, in keeping with Gibson's theory. 
And it demonstrates a fundamental 
advantage of virtual reality: it facili- 
tates active perception and explo- 
ration of the environment portrayed. 

Anthropologist Lucy Suchman 
(1988) has said, "Every human tool 
relies upon, and deifies, some under- 
lying conception of the activity that it 
is designed to support, as a conse- 
quence, one way to view the artifact is 
as a test on the limits of the underly- 
ing conception" (p. 3). The underly- 
ing conception is based upon the 
affordances the tool offers. Virtual 
reality is no exception. We expect 
that when the representation or 
avatar of our hand encounters a wall, 
it will be afforded resistance; we don't 
expect to be able to walk through 
walls. Since virtual reality is a repre- 
sentation of some reality, we expect it 
to offer the same, or at least 
comparable, affordances as the real 
world it imitates. 

Stephenson's science fiction 
novel Snow Crash provides the most 
intriguing examination of design 
issues related to affordances in virtual 
reality to date. In Stephenson's story, 
the main character, Hiro Protagonist 
is a man of many talents. He is "the 
last of the freelance hackers." He is 
also "the greatest sword fighter in the 
world." In addition, he gathers and 
sells information. His avatar in the 
virtual world of the Metaverse is 
represented with swords. Hiro 
carries swords in real life, too; they 
are family heirlooms. What happens 
in virtual reality when there is a 



sword fight and someone's virtual 
legs are hacked off? This presents a 
design problem. Stephenson de- 
scribes the aftermath of a sword fight 

The [man] lies cut in segments 
on The Black Sun's floor. 
Surprisingly (he look: so real 
when he's in one piece), no 
flesh, blood, or organs are 
visible through the new cross 
sections that Hiro's sword 
made through his body. He is 
nothing more than a thin shell 
of epidermis, an incredibly 
complex inflatable doll. But 
the air does not rush out of 
him, he fails to collapse, and 
you can look into the aperture 
of a sword cut and see, instead 
of bones and meat, the back of 
the skin on the other side. It 
breaks the metaphor. The 
avatar is not acting like a real 
body. It reminds all The Black 
Sun's patrons that they are 
living in a fantasy world. 
People hate to be reminded of 
this (p. 103). 

So what happens to restore the 
metaphor, so that it meets the virtual 
traveler's expectations in terms of 
affordances? Stephenson explains: 

...the first thing that happens, 
when someone loses a sword 
fight, is that his computer gets 
disconnected from the global 
network that is the Metaverse. 
He gets chucked right out of 
the system. It is the closest 
simulation to death that the 
Metaverse can offer, but all it 
really does is cause the user a 
lot of annoyance. 
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Furthermore, the user finds 
that he can't get back into the 
Metaverse for a few minutes. 
He can't log back on. This is 
because his avatar, dismem- 
bered, is still in the Metaverse, 
and it's a rule that your avatar 
can't exist in two places at 
once. So the user can't get 
back in until his avatar has 
been disposed of. 

Disposal of hacked-up avatar 
is taken care of by Graveyard 
Daemons, a new Metaverso 
feature that Hiro had to invent. 
They are small lithe persons 
swathed in black, like xunjas, 
not even their eyes showing. 
They are quiet and efficient. 
Even as Hiro is stepping back 
from the hacked-up body of 
his former opponent, they are 
emerging from invisible trap- 
doors in The Black Sun's floor, 
climbing up out of the nether 
world, converging on the 
fallen businessman. Within 
seconds, they have stashed the 
body parts into black bags. 
Then they climb back down 
through their secret trapdoors 
and vanish into hidden tunnels 
beneath The Black Sun's floor 
(p. 102-103). 

This is an elaborate and fanciful 
vision of how continuity and affor- 
dances can be restored in virtual 
reality. However, there are already 
some comparable situations in 
existing virtual reality systems where 
the design metaphor must be 
readjusted. For example, in BattleTech 
- a VR entertainment system for 
multiple players - players who lose 



their vehicles to enemy firepower get 
wafted to a new one so they can play 
out their game time, even after they 
crash. This is a way of upholding the 
metaphor and maintaining continuity 
while at the same time giving players 
their full money's worth. The players 
pay for each game, so they get to play 
out the simulation during the entire 
time they have paid for, even if they 
lose a round. 

In his more recent work on VR 
as a form of scientific visualization for 
planetary exploration, Michael 
McGreevy (1993) of the NASA Ames 
Lab has taken up the Gibsonian idea 
that the environment must "afford" 
exploration in order for people to 
make sense of it. McGreevy has 
linked Gibson's concept of affordance 
to the idea that people can begin to 
learn something important from the 
data they have retrieved from 
planetary exploration by flying their 
points of view through the images 
themselves. McGreevy (1993) ex- 
plains: "Environments afford explo- 
ration. Environments are composed 
of openings, paths, steps, and shallow 
slopes, which afford locomotion. 
Environments also consist of obsta- 
cles, which afford collision and 
possible injury; water, fire, and wind, 
which afford life and danger; and 
shelters, which afford protection from 
hostile elements. Most importantly, 
environments afford a context for 
interaction with a collection of 
objects" (p. 87). As for objects, they 
afford "grasping, throwing, 
portability, containment, and sitting 
on. Objects afford shaping, molding, 
manufacture, stacking, piling, and 
building. Some objects afford eating. 
Some very special objects afford use 



as tools, or spontaneous action and 
interaction (that is, some objects are 
other animals)" (p. 87). 

McGreevy (1993) points out that 
natural objects and environments 
offer far more opportunity for use, 
interaction, manipulation, and explo- 
ration than the ones typically 
generated on computer systems. 
Furthermore, a user's natural capacity 
for visual, manipulative, and 
locomotor interaction with real envi- 
ronments and objects is far more 
informative than the typically 
restricted interactions with computer- 
generated scenes. A virtual world 
may differ from the real world, but 
virtual objects and environments 
must provide some measure of the 
affordances of the objects and envi- 
ronments depicted (standing in for 
the real world) in order to support 
natural vision more fully. 



Interfaces 

Let's now turn to the third 
design topic, interfaces. Interfaces for 
both input and output, link the user 
to the virtual world which exists only 
digitally. Interface devices such as 
the glove and the full-body input 
devices offer far more than the 
convenience of being able to 
manipulate virtual objects by 
reaching out and picking them up. So 
the image of a hand in a virtual world 
is an affordance and an avatar, linked 
directly to an interface device: the 
wired glove. Interface devices such 
as the wired glove are very important 
for creating a sense of presence and 
establishing a feel for the spatial 
dimensions of the virtual world. 



According to Ellis (1992), a large 
part of our sense of physical reality is 
a consequence of internal processing 
rather than being something that is 
developed only from the immediate 
sensory information we receive. Our 
sensory and cognitive interpretive 
systems are predisposed to process 
incoming information in ways that 
normally result in a correct 
interpretation of the external 
environment, and in some cases they 
may be said to actually "resonate" 
with specific patterns of input that are 
uniquely informative about our 
environment 

These same constructive pro- 
cesses are triggered by the interfaces 
used to present virtual environments. 
However, in these cases the informa- 
tion is mediated through the inter- 
faces, including the display technol- 
ogy. The illusion of an enveloping 
environment depends on the extent to 
which all of these constructive 
processes are triggered. This is 
related to the continuity of the affor- 
dances between the real world and its 
electronic virtual surrogate. This 
includes the options we have for 
interfacing with the virtual world, 
and these options have serious limita- 
tions, as Brenda Laurel (1992) points 
out. 

According to Laurel, there is a 
serious navigation problem n virtual 
reality, limiting how designers can 
give people the sense that they're 
moving through large virtual spaces. 
The most common way to do this is 
by pointing a finger enclosed in a 
wired glove and "finger flying." In 
the real world, we walk, we don't 
point a finger and expect to move. 



Pointing and flying don't give the 
user the kinesthetic satisfaction of 
being able to move all over the place. 
Some developments, such as Fake 
Space Labs 9 BOOM visual display and 
the University of North Carolina's 
treadmill, work toward providing 
partial solutions to this problem, but 
a fully adequate solution remains to 
be discovered. No one has yet come 
up with the winning answer to the 
question of virtual locomotion. 

Laurel points up another impor- 
tant issue related to multisensory 
interfaces in virtual reality: "sensory 
combinatorics." According to Laurel, 
"We're working in a multisensory 
medium: we're trying to cram as 
many senses as we can into 
telepresence and virtual environ* 
ments. How do these senses interact 
with each other? How do you design 
for this medium?" (p. 291). This 
notion of sensory combinatorics, then, 
concerns putting sensors together 
properly in a multisensory 
environment. 

One example of the design 
potential of combining sensory 
interfaces concerns the role of audio 
in virtual reality. Laurel explains that 
audio affects the perception of video: 
"In the computer game business, we 
learned that high-resolution audio 
caused people to report that the 
game's graphics were higher 
resolution. It doesn't work the other 
way though; really high-resolution 
video does not cause people to report 
that the little beeps on their PC are 
suddenly rich, full-bodied sounds. 
Something interesting is going on 
here. In one case, audio pulls video 



along, and m the other, a disparity 
shatters the illusion" (p. 289). 

According to Laurel (1992), 
"audio pulls us along in a dimension 
that we might call "involvement," or 
the dimension of "constructive 
involvement": pulling stuff out of the 
imagination of your user to achieve a 
very deep level of participation. It 
turns out that audio is a whole lot 
better at that, in general, than video" 
(p. 290). When you create an audio- 
intensive environment, the effect 
tends to work better, although there 
is no guarantee. Laurel advises that 
three-dimensional audio is much 
better than video at creating a sense 
of real space. 

Conclusion 

This paper has focused on three 
important design issues in virtual 
reality: avatars, affordances, and 
interfaces. As we have seen, these 
three design themes are closely inter- 
related. They deal with the changing 
inter-relationship between reality and 
representation that virtual reality 
compels us to deal with. Every 
technology has forced a reconsidera- 
tion of this issue, but none more 
profoundly titan virtual reality. 

In exploring these design 
themes, the focus has been upon 
examples, including fictional 
examples from the science fiction 
novel Snow Crash. Design in virtual 
reality is almost completely new. In 
thinking about design for this new 
medium, past technologies must be 
considered, together with new 
visions. Jerome Bruner (1990) 



10 



proposed a bipolar model of story- 
telling: paradigmatic storytelling 
("just the facts") and narrative story- 
telling. Designers exploring the 
potential of the new medium, virtual 
reality, need to consider both kinds of 
storytelling in order to get the 
technical specifications right while at 
the same time understanding what 
the virtual world really feels like to 
the user. Neal Stephenson has 
referred to this second kind of 
understanding as "condensing fact 
from the vapor of nuance." 
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